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To  summarize,  the  methodology  for  objective  performance  measurement  of  combat  system 
effectiveness  depends  on  human  factors  experimentation  in  a  simulated  tactical  environment  based 
on  decision  analysis.  It  features  a  rapid  prototyping  capability  and  performance  assessment  system. 
In  the  short  term,  these  methods  will  help  designers  avoid  system-induced  human  errors  and 
inefficiencies.  Eventual  application  should  provide  system  capabilities  for  mitigating  human 
information  handling  limitations  and  biases.  The  overall  result  should  be  submarine  combat  systems 
that  enhance  platform  tactical  effectiveness  through  the  full  range  of  operations  in  a  more  affordable 
manner. 
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Simulation  Facility  Development 


The  development  of  a  simulation  environment  to  support  the  experimental  design  has  been 
undertaken.  It  consists  of  a  simulation  manager  that  creates  and  maintains  a  medium-fidelity  tactical 
environment,  making  scenario  data  available  to  other  components.  The  experimenter  controls 
vignette  interaction  from  this  station.  The  second  component  is  the  command  workstation,  which 
provides  the  means  for  interaction  of  the  subject  with  proposed  displays.  A  large  screen  display  has 
also  been  installed  for  additional  capability.  An  integral  decision-aid  station  has  been  provided  to 
investigate  advanced  decision-aiding  concepts  within  the  tactical  simulation  environment. 

A  key  component  in  the  system  is  the  display  authoring  system,  currently  under 
development.  It  provides  a  rapid  display  prototyping  capability  that  allows  display  building  by 
non-programmers  and,  potentially,  subject  matter  experts.  Candidate  displays  are  configured  here 
for  immediate  insertion  into  the  connmand  workstation  where  they  can  immediately  interact  with  the 
scenario  running  in  the  simulation  manager.  A  performance  assessment  system  is  also  being 
developed  to  support  the  defined  MOE/MOPs  and  to  offer  as  much  of  an  automated  data  collection 
system  for  experimental  purposes  as  desired.  Future  improvements  include  incorporation  of  a  3-D 
graphics  system.  A  detailed  description  of  the  <^acility  appears  in  a  previous  report  (Funk,  Kaiwi,  & 
Fernandes,  1988). 


SUMMARY 

The  scope  of  our  initial  effort  has  broadened  to  include  much  more  than  objective 
performance  measurement  of  submarine  combat  system  features.  A  byproduct  of  the  top-down 
analysis  of  the  CO's  tasks  includes  potential  knowledge-based  guidelines  for  the  use  of  the  display 
authoring  system  and  system  design.  Guidelines  distilled  from  specific  task  analysis  form  the  basis 
for  using  the  rapid  prototyping  capability  by  focusing  on  explicit  user-centered  issues. 

The  decision  analysis  conducted  at  the  data-specific  level  can  also  benefit  training.  A  great 
deal  of  time  and  effort  is  expended  in  generating  exercise  controller  guides  for  tactical  team  training. 
These  guides  consist  primarily  of  the  target  geometries  to  be  used  during  the  training  scenarios.  As 
noted  earlier,  given  how  long  the  CO  has  to  interact  with  the  initial  target  geometry,  the  final 
problem  configuration  bears  little  resemblance  to  that  provided  in  the  exercise  controller  guide.  It  is 
the  instructor  who  creates  and  maintains  tactical  geometries  in  this  interactive  environment,  actions 
which  are  by  and  large  not  specified  or  documented.  Examining  the  tactical  data  and  its  use  by  the 
CO  may  lead  to  more  efficient  training.  This  has  been  discussed  with  the  Tactical  Training 
Department  of  the  Submarine  Training  Facility,  San  Diego,  California. 

The  level  of  abstraction  in  tactical  data  presentation  may  also  hold  insights  for  decision-aiding 
capabilities.  For  instance,  it  might  be  argued  that  a  pictorial  situation  assessment  display  should  be 
favored  over  an  alphanumeric  display  for  ease  of  interpretation.  Consider,  however,  that  the 
sensitivity  of  recognition  to  a  parameter  such  as  bearing  rate  is  enhanced  by  the  abstraction  of 
alphanumeric  presentation. 

The  analysis  of  tactical  data  also  brought  to  light  the  generic  nature  of  contact  data  from 
completely  different  sensor  types.  Aggregation  of  tactical  data  into  spatial  or  object  categories  also 
holds  promise  for  further  work  in  decision-aiding  concepts.  A  report  addressing  specific  decision 
aids  has  already  been  published,  with  another  in  preparation.  The  next  report  in  this  effort  will  be  a 
compilation  of  all  the  specific  vignettes  and  the  methodology  in  their  development. 
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Vignette  Development 

The  evolution  of  the  target  engagement  vignettes  has  progressed  from  our  initial  analysis. 
The  generic  steps  in  each  of  the  final  vignettes  have  already  been  defined.  Further  definition, 
presently  underway,  will  concern  the  CO  decision  sequence  within  each  step  of  the  vignette,  and 
will  take  the  form  of  a  script.  The  intent  is  to  "decompose"  the  overall  decision  sequence  into  its 
component  parts  in  order  to  specify  the  uore  considerations  from  the  CO's  perspective.  The  purpose 
is  not  to  model  CO  behavior,  bjL  to  provide  a  framework  for  vignette  development.  By  using 
man-in-the-loop  experimentation,  the  CO  brings  his  experience  and  knowledge  regarding  target  and 
weapon  capabilities  to  the  experimental  environment  without  the  experimenters  having  to  model  it. 

Specific  information  sources  and  types  of  data  required  by  the  CO  to  complete  each  portion 
of  the  vignette  decioion  sequence  will  then  be  identified.  Specific  decision  rules  and  evaluation 
criteria  used  by  the  CO  to  relate  information  input  to  decision  output  in  each  sequence  can  also  be 
identified,  as  well  as  the  data  set  used  for  each  decision  sequence,  including  logical  linkages  to  prior 
and  succeeding  vignettes  (see  Figure  2).  It  is  this  data-specific  approach  to  scenario  development 
that  distinguishes  these  vignettes  from  previous  scenario-building  efforts. 


A  tactical  vignette  is  a  set  of  operational  events  (including  CO  decisions  and  actions)  that 
occur  in  chronological  sequence.  A  set  of  initial  conditions  sets  the  stage  for  the  vignette.  We  will 
use  sets  of  initial  conditions  to  define  tactical  situations  that  examine  variations  of  the  decision 
sequences  involved  and  yield  the  most  useful  measures  of  performance.  Sets  of  initial  conditions 
and  events  for  each  specific  vignette  will  then  be  categorized.  The  completed  vignettes  will  form  a 
cohesive  package  of  initial  conditions,  events,  outcomes,  and  MOE/MOPs.  These  vignettes,  then, 
are  not  experiments  but  the  experimental  tactical  environment  for  use  in  objective  performance 
measurement. 
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Only  portions  of  the  overall  decision-making  process  exhibited  by  the  CO  are  amenable  to 
decision  path  analysis.  This  is  due  to  the  modular  nature  of  decisions  in  a  data-driven,  real-time 
environment.  Certain  decisions  must  be  deferred  until  enough  data  is  available  for  their 
consideration.  Additionally,  certain  decisions  may  have  to  be  made  more  than  once  depending  on 
tactical  considerations.  Recent  experiments  indicate  that  the  segmenting  of  decisions  on  the  pan  of 
the  CO  may  be  a  common  decision-making  strategy. 

The  two  major  parallel  processes  engaging  the  CO  with  respect  to  sonar  target  prosecution 
are  localization  and  classification.  These  are  continuous  processes,  characterized  by  "state"  updates 
with  respect  to  target  parameters.  For  instance,  while  the  ship  is  localizing  a  sonar  contact,  the  range 
to  that  contact  will  change  in  a  continuous  manner,  passing  through  certain  "gates"  for  applicable 
sensor  and  weapon  employment.  As  those  gates  are  passed,  the  state  is  incrementally  updated  by 
the  CO  and  held  until  another  gate  is  passed.  These  gates  are  easily  employed  as  specific  initial 
conditions  or  events  during  the  tactical  vignettes. 

By  segmenting  the  CO  decision  process  into  decision  vignettes,  it  might  be  argued  that  a  lack 
of  process  continuity  would  adversely  impact  experimental  results.  Consider,  however,  that  each 
separate  decision  sequence  has  its  own  initial  conditions  and  categories  of  outcomes.  The  outcome 
of  one  decision  sequence  becomes  the  initial  condition  for  the  following  decision  vignette,  linking 
the  separate  vignettes  in  terms  of  context,  if  not  time.  An  error  by  a  CO  resulting  in  an  adverse 
outcome  in  one  vignette  is  represented  by  unfavorable  initial  conditions  in  one  or  more  following 
vignettes. 

Measures  of  Performance 

The  measures  of  performance  and  measures  of  effectiveness  (MOPs/MOEs)  developed  to 
evaluate  quality  and  timeliness  of  selected  command  decisions  focus  on  three  areas:  time,  accuracy, 
and  adequacy.  Elapsed  time  for  specific  decision  events  is  quite  a  valuable  measure,  particularly  in  a 
vignette  restricted  to  several  minutes  in  length.  Accuracy  pertains  to  the  number  and  types  of 
mistakes  made  in  a  given  scenario.  Finally,  since  there  may  be  multiple  adequate  solutions  in  a 
dynamic  tactical  environment,  our  effort  will  be  directed  at  identifying  those  combat  system  features 
that  allow  the  CO  to  pick  the  best  one. 

The  MOE/MOPs  identified  are  intermediate  rather  than  ultimate  performance  criteria.  The 
intent  was  to  focus  on  observable  events.  The  MOEs  for  given  tactical  scenarios  tend  to  evolve  into 
a  hierarchy,  as  detailed  by  another  researcher  (Rau,  1974).  In  addition  to  the  MOP/M OEs  currently 
identified,  others  that  are  display-  or  experiment-specific  will  undoubtedly  develop.  Again,  the 
purpose  is  not  to  propose  why  mistakes  are  made,  but  to  determine  which  displays  tend  to  give 
better  performance.  In  a  sense  we  are  developing  relative  MOPs  vice  absolute  MOPs  owing  to  the 
part-task  approach  to  this  effort.  We  are  evaluating  displays,  not  COs. 

As  stated  above,  additional  MOPs  will  develop  through  refinement  of  the  experimental 
methodology.  For  instance,  the  two  modes  of  FCP  failure  can  be  characterized  in  two  ways.  First, 
there  are  those  s.^aations  that  are  too  dynamic  for  the  FCP  architecture,  with  its  overriding 
communications  requirements.  Secondly,  failures  can  occur  due  to  unexpected  conditions  or  target 
actions  that  do  not  fit  into  the  CO's  pre-programmed  responses.  The  observation  of  different  types 
of  coping  behavior  exhibited  by  COs  in  a  tactical  environment  might  also  produce  some  additional 
MOPs  (Foley  &  Wallace,  1974;  Sage,  1981). 
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APPROACH 


The  development  of  techniques  to  objectively  evaluate  combat  system  displays  has  proceeded 
along  two  parallel  paths  at  NOSC.  First,  a  progressive  functional  analysis  for  experimental  design 
was  conducted,  yielding  the  methodology  of  investigation.  Secondly,  and  concurrently,  the 
development  of  an  experimental  tactical  environment  for  implementing  performance  measurement 
was  undertaken. 

We  spent  considerable  time  in  our  early  efforts  on  problem  definition  and  experimental 
design.  A  functional  analysis  of  the  approach  and  attack  scenario  led  to  an  outline  of  technical  steps 
required  to  improve  information  integration  in  the  submarine  combat  system.  A  technical  analysis 
was  undertaken  to  investigate  current  combat  control  procedures.  This  consisted  of  document 
review,  combat  team  observation,  and  interviews  and  workshops  with  experts. 

Document  review  resulted  in  a  collation  of  current  tactical  doctrine  and  practices  from  a 
diverse  selection  of  publications.  Observation  of  combat  teams  in  the  training  environment  allowed 
investigators  to  confer  with  instructional  staff  familiar  with  tactical  evaluation  and  to  question  COs 
during  the  performance  of  their  duties.  All  these  efforts  led  to  a  comprehensive  understanding  of  the 
problem  environment. 

In  order  to  identify  key  decisions  and  actions  performed  by  the  CO  during  target  encounters, 
a  formal  descriptive  model  was  developed.  This  model  identified  significant  decisions  and  actions, 
events,  and  informational  inputs  that  impact  these  decisions  and  actions,  and  general  sequencing  of 
them  during  tactical  evolutions.  The  resulting  descriptive  decision  model  was  scenario-independent 
so  that  it  would  not  simply  be  a  time-based  narrative  chain  of  events.  It  was,  however,  reflective  of 
the  dynamic  real-world  environment,  with  multiple  parallel  activities  rather  than  idealized  sequential 
phases. 

The  Experimental  Environment 

A  target  encounter  scenario  in  submarine  warfare  can  vary  from  .5  to  4  hours  in  length. 
There  are  roughly  25  people  in  the  FCP.  This  kind  of  real-time  interactive  environment  is  difficult  to 
analyze  from  an  operability  standpoint  because  it  tends  not  to  be  repeatable.  In  order  to  provide  an 
elem.ent  of  experimental  repeatability,  several  things  were  considered.  First,  as  noted  earlier,  the 
submarine  environment  is  less  dynamic  and  less  complex  than  other  warfare  areas  and  made  a  good 
starting  point  for  initial  analysis.  Next,  by  restricting  the  experimental  environment  to  the  CO 
exclusively,  we  eliminated  the  confusion  of  having  many  people  to  keep  track  of  during  an 
experiment.  Finally,  we  restricted  the  time  available  for  interaction  to  a  slice  of  time  only  several 
minutes  in  length  around  a  specific  decision  that  the  CO  must  make.  These  three  considerations 
formed  the  basis  of  the  part-task,  medium-fidelity  simulation  environment  to  be  used  for 
experimentation. 

The  validity  of  a  part-task  experimental  environment  is  crucial  to  the  applicability  of  our 
experimental  results.  It  must  be  emphasized  that  the  purpose  of  this  effort  is  to  determine  display 
utility  for  a  given  task  or  range  of  tasks,  since  no  single  display  or  decision  aid  will  support  all  CO 
decision-m^ing  requirements.  It  made  sense,  therefore,  to  segment  the  overall  target  engagement 
scenario  into  the  slices  of  time,  or  vignettes,  about  critical  decisions  for  which  a  particular  display  or 
decision  aid  was  targeted. 


3 


BACKGROUND 


The  submarine  platform  represents  a  microcosm  of  the  larger  command  and  control 
environment.  It  is  felt  ^at  initial  analysis  undertaken  in  this  area  will  be  manageable  due  to  the 
nature  of  the  tactical  environment  This  environment  is  characterized  by  relatively  slow  speeds  and 
fewer  contacts  due  to  the  types  of  organic  sensors.  Additionally,  the  communications  requirements 
for  submarines  are  much  less  stringent  than  those  for  other  platform  types,  and  the  functional 
command  structure  is  not  distributed  among  several  people.  It  is  more  or  less  a  pure  platform-level 
environment.  The  tactical  environment  is  getting  much  more  demanding  due  to  the  erosion  of  the 
technological  advantage  enjoyed  for  so  long  by  the  submarine  force.  Detection  ranges  have 
fundamentally  changed  system  and  operator  requirements.  This  makes  for  a  critical  opportunity  to 
improve  combat  system  interface  utility. 

The  incorporation  of  a  specific  command  interface  in  a  submarine  combat  system  introduces  a 
new  and  powerful  capability.  No  other  single  person  on  the  submarine  contributes  more  to  the 
tactical  effectiveness  of  the  platform  than  the  Commanding  Officer  (CO).  Historically,  the  CO  has 
been  isolated  from  tactical  data  by  a  large  number  of  operators  and  supervisors  comprising  the  Fire 
Control  Party  (FCP).  These  intermediate  personnel  serve  as  selective  filters  and  buffers.  In 
previous  systems  the  designated  command  displays  were,  by  and  large,  underutilized  by  the  CO 
because  they  were  either  inaccessible  or  cumbersome.  The  use  of  an  effective  command  interface 
and  decision  aids  may  be  able  to  bridge  the  gap  in  the  approach  and  attack  architecture,  giving  the 
CO  timely  access  to  the  relevant  tactical  data  he  needs  to  do  his  job.  Additionally,  the  combat  system 
should  provide  better  spatial  orientation  and  correlation  of  data  to  everyone  in  the  FCP. 

Initial  efforts  at  measuring  tactical  performance  of  submarines  were  aimed  at  recording 
tactical  parameters  in  a  training  environment  (Callan,  Kelly,  &  Nicotra,  1978).  These  procedures 
were  expanded  and  formalized  with  incorporation  of  the  Submarine  Advanced  Reactive  Tactical 
Training  System  (SMARTTS)  in  the  design  evaluation  of  digital  combat  systems  (Ship  Analytics, 
1983).  This  approach  designated  certain  tactical  parameters  as  "performance  indicators,"  though 
justification  of  these  designations  was  never  fully  specified  or  developed.  The  SMARTTS  capability 
was  not  meant  to  be  comprehensive.  It  was  recognized  at  that  time  by  the  researchers  that 
interpretation  of  recorded  tactical  parameters  still  required  the  subjective  assessments  of  instructors 
or  subject  matter  experts  to  be  useful.  Their  approach  was  never  improved  to  a  sufficient  level  of 
detail  or  completeness  to  support  NOSC's  efforts  in  performance  measurement. 

A  decision-oriented  approach  to  defining  combat  system  data  requirements  in  support  of 
design  efforts  for  the  next-generation  combat  system  was  conducted  a  few  years  later  at  the  Naval 
Underwater  Systems  Center  (NUSC).  While  comprehensive,  this  decision  analysis  was  still  at  a 
level  of  detail  insufficient  to  support  experimental  design  and  evaluation  of  combat  system 
interfaces.  A  series  of  "concept  of  operations  experiments"  (COOPEXs)  was  initiated  by  NUSC  to 
augment  the  information  management  analysis.  These,  however,  were  severely  restricted  by 
hardware  capabilities  that  could  not  support  real-time  tactical  scenarios  and  took  the  form  of 
seminars  and  questionnaires  rather  than  experiments. 

Unfortunately,  measures  of  performance  historically  used  for  systems  evaluation  or 
wargaming  have  taken  the  form  of  complex  products  of  probabilities  with  little  direct  application  to 
individual  decision-making  performance  (Rau,  1974).  These  were  also  not  usable  in  an 
experimental  environment  aimed  at  display  ev^uation. 
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INTRODUCTION 


This  project  investigc.tes  man-machine  interface  issues  and  operability  enhancements  for  use 
in  the  design  of  combat  systems  for  future  attack  submarines.  The  Naval  Ocean  Systems  Center 
(NOSC)  will  design  and  conduct  experiments  concerned  with  submarine  combat  control  and 
decision  making.  A  top-down  decision  analysis  forms  the  basis  of  experimental  design  within  a 
part-task,  medium-fidelity  simulated  tactical  environment.  Rapid  prototyping  of  proposed  displays 
and  decision  aids  that  can  be  immediately  linked  with  this  simulation  will  provide  a  timely  and 
cost-effective  vehicle  for  objectively  evaluating  display  utility  from  the  user's  perspective. 

The  objective  of  these  experiments  wUl  be  to  evaluate  system  design  and  procedural  factors, 
such  as  display  formats,  operational  routines,  and  decision  aids.  The  methods,  procedures,  and 
facilities  for  designing  an  operable  combat  system  resulting  from  this  effort  wiU  be  incorporated  into 
combat  system  acquisition  programs  as  a  transportable  product  available  for  use  by  government  and 
contractor  personnel  involved  in  specification,  design,  and  evaluation  of  new  combat  systems. 

The  ability  of  a  performance-based  evaluation  system  to  provide  a  try-before-you-buy 
capability  for  complex  combat  system-human  interfaces  can  yield  several  other  important  benefits. 
The  identification  of  system  software  shortfalls  early  in  the  design  process  should  save  on  expensive 
software  revisions.  This  identification  may  also  drive  some  hardware  design  considerations.  The 
overall  utility  of  the  final  system  should  also  decrease  training  requirements  that  are  costly, 
time-consuming,  and  difficult  by  reducing  operator  hands-on  dme  required  for  proficiency. 

The  ultimate  goal  of  this  effort  is  to  replace  subjective  preference  with  objective 
man-in-the-loop  performance  measurement  for  evaluating  interface  utility.  Experimental  results  will 
provide  the  character  and  quantification  of  benefits  to  be  achieved  in  improved  designs.  Finally, 
objective  examination  of  potential,  sophisticated  combat  system  features  will  ultimately  yield  a  more 
capable  platform.  Figure  1  is  included  here  to  guide  the  reader  through  the  report. 


Figure  1.  Development  of  performance  measurement  methodology  for  combat  system  operability. 


1 


Performance  Measurement  Methodology  for  Enhanced  Submarine 
Combat  System  Effectiveness 


Clement  D.  Urban 
Naval  Ocean  Systems  Center 
Information  and  Decision  Management  Branch 
Code  442 
27 1  Catalina  Blvd. 

San  Diego,  CA  92152-5000 
(619)  553-9223  AUTOVON  553-9223 
e-mail:  curban  @  nosc.mil 


ABSTRACT 

This  presentation  outlines  the  development  of  a  methodology  for  objective  man-in-the-loop 
performance  measurement  of  combat  system  features  based  on  experimentation  in  a  simulated 
tactical  environment.  A  history  of  submarine  tactical  performance  measurement  is  discussed.  First, 
a  top-down  functional  analysis  for  experimental  design  was  conducted  yielding  a  descriptive 
decision  model.  Considerations  involving  experimental  repeatability  and  the  validity  of  results  in  a 
part-task,  medium-fidelity  environm.ent  are  discussed  as  well  as  measures  of  performance.  A 
data-specific  approach  to  decision  analysis  is  described  which  distinguishes  this  effort  from  other 
scenario  building  efforts.  Implications  of  this  approach  for  decision  aiding  and  training  efforts  are 
discussed.  In  conjunction  with  analytical  efforts,  the  development  of  a  simulation  facility 
incorporating  rapid  prototyping  is  presented. 
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